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Abstract 

Constructs underlying student ratings of 
instruction in an introductory master's level 
educational research course were investigated in order 
to gain better understanding of instructor behaviors 
which facilitate student success • During academic 
years 1990-91 and 1991-92, an institutionally developed 
scale was administered to students enrolled in master's 
level education courses at a metropolitan university • 
Included among 1597 student ratings were 144 from 
students who rated research methods classes • 
Independent principal components analyses followed by 
varimax rotations were conducted on ratings from those 
enrolled in research methods and those enrolled in 
other graduate classes. Research methods ratings 
yielded four factors accounting for 75.2% of variance 
while from other graduate education ratings only three 
factors accounted for 73.3% of variance. The unique 
research factor "Thoroughness" together with specific 
item comparisons suggested the existence of a different 
rating policy within research methods classes. 
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Introduction 

Student ratings of instruction have long been a 
major source of data for evaluating college faculty and 
courses. Student rating results are extensively used 
for both formative and summative evaluation. Research 
on the student rating process and its components has 
been abundant in literature in the field of faculty 
evaluation (Arreola and Aleamoni, 1990). Much of the 
research on student ratings has focused on teacher 
characteristics used by students to evaluate 
instruction (Banz and Rodgers, 1985 p. 267) . Knowledge 
of such characteristics can be used by faculty to plan 
instructional strategies designed to enhance student 
motivation and learning. 

Numerous factor analytic studies have concluded 
that student rating forms generally measure multiple 
aspects of teaching (Marsh and Hocevar, 1984 p. 
342 - 43). The factor structure of an instrument 
provides evaluators with knowledge of dimensions on 
which students base judgments about the instructor and 
the teaching-learning process. Such knowledge can 
obviously be of use to those who desire to improve 
student outcomes in the courses they teach. 
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Abrami (1989) emphasizes that the dimensions of 
student ratings which emerge from analysis of the 
factor structure of rating instruments lack 
generalizability across a variety of course, 
instructor, and student settings, Abrami, d'Apollonia, 
and Cohen (1990) stress the importance of studying 
student ratings in the local institutional context. 
Identification of rating dimensions distinctively 
associated with specific courses could shed 
considerable light on peculiar attributes of the 
teaching learning process as it pertains to those 
courses. 

Faculty who teach introductory graduate courses in 
educational research frequently observe that their 
students appear to experience considerably higher 
levels of course related anxiety and discomfort than is 
the case with other graduate level education courses. 
They generally speculate that this phenomenon is 
related to the quantitative nature of the course and 
the fact that students are often encouraged or required 
to take research methods as their first graduate 
course . 

Several studies have examined attitudes of 
graduate students in education and psychology toward 
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research and research instruction. Royalty, et al. 
(1986) investigated counseling psychology students' 
attitudes toward research training in doctoral degree 
programs. Subjects were found to have less than 
positive attitudes toward research and research 
training even at the time they entered doctoral 
programs. Benton and Jerrolds (1982) studied the 
relationship between research attitude and achievement 
in an introductory master's level educational research 
course. While attitudes were found to have improved by 
the end of the course, no correlates of attitude 
improvement were identified nor was attitude found to 
be significantly related to achievement. 

The purpose of this study was to identify 
distinctive aspects of student rating patterns in an 
introductory master's level educational research 
methods course in order to gain better understanding of 
instructor behaviors which might facilitate student 
success in the course. Principal components analysis 
was used to compare constructs that students used to 
evaluate instruction in the research course with those 
used to evaluate instruction in other graduate 
education courses . 
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Method 

Subjects 

Sub j ects of this study were master ' s level 
students enrolled in education courses at a 
metropolitan university in Tennessee. Included among 
subjects were students majoring in education, 
counsel ing , and school psychology . Subj ects 

participated in the university's student rating of 
faculty instrurrtion process during academic years 
1990 - 91 and 1991 - 92. Ea^.h university class taught 
during the fall semester is routinely rated by all 
enrolled students. Courses not taught in the fall and 
class sections of fall rated courses taught by other 
instructors are rated during spring and summer terms . 
A total 1597 ratings were received from graduate 
education classes rated during the two academic years. 
One hundred forty-five of those ratings were obtained 
from seven sections of the educational research methods 
course taught by four different instructors. 
Instrui::^ent 

A seventeen-item institutionally-developed seven- 
point likert-type rating scale was used to elicit 
perceptions of instructor and course characteristics 
thought to be related to instructional outcomes. An 
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Alpha reliability coefficient of .96 was obtained from 
administration of the instrument across all levels of 
classes and academic divisions within the university. 
Factor analyses conducted on previous results of the 
rating instrument have generally identified three 
non-trivial factors at all course levels and within 
several disciplines. 
Procedure 

Separate principal components analyses were 
conducted on the ratings obtained from two groups of 
subjects: those enrolled in sections of the research 
methods course and those enrolled in other graduate 
education classes. Factors with eigenvalues greater 
than one were selected for varimax rotation. 
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Results 

Four factors with eigenvalues greater than one 
accounted for 75.2% of the variance within the research 
methods rating data. From the ratings of other 
graduate education classes only three factors emerged 
accounting for 73.3% of the variance. 
Factor Analysis of Non-Research Ratings 

Table 1 displays ratings items and rotated factor 
loadings resulting from principal components analysis 
of ratings obtained from the other (non-research) 
graduate education classes. The first factor which 
accounted for 59.1% of the variance loaded on three 
general items: 16 (overall teaching effectiveness), 15 
(perceived student learning) , 17 (desire to take 
another course from the instructor) , and seven specific 
competency related items. This factor was named 
"competency. " 

Factors II and III accounted respectively for an 
additional 7.8% and 6.7% of variance among items rated 
by students in the non-research classes . Factor 1 1 
loaded on four rating items dealing with relevance, 
fairness, and communication of class evaluation and 
assignments. This factor was named "teaching equity." 
The three items on which Factor III loaded elicit 
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student perceptions of the extent to which the 
instructor is helpful and respectful to students. This 
factor was labeled "interpersonal skills," 
Factor Analysis of Research Ratings 

Table 2 presents results of the principal 
components analy s is perf ormed on rat ings obta ined from 
the research methods classes. As was true with the 
non-research group, Factor I appears to be a competency 
construct loading on eight of the ten items that 
comprised "competency" among the non-research ratings 
and accounts for 49.5% of variance. Two items however 
which loaded on non-research Factor I , item 1 (clear 
understanding of topics) and item 7 (makes students 
aware of objectives) failed to load even moderately on 
the first factor extracted from the research data. 
Item 10 (helpful feedback) loaded highest on research 
Factor I even though it continued to load moderately on 
the third factor. 

Factor II which accounted for 12 . 1% of variance, 
shared three common items with the second construct 
extracted for the non-research ratings: 11 (relevant 
assignments and exams) , 12 (grades fairly) , and 13 
(keeps students informed of progress) . Two competency 
items, 1 (clear understanding of topics) and 7 (makes 
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students aware of objectives) also loaded on Factor II 
which was named "instructional clarity." 

Research Factor III included two of the three 
items that loaded on its non-research counterpart: 9 
(respects students) and 8 (willing to assist students) . 
This factor accounted for 7.5% of variance and was 
again labeled " interpersonal skills . " 

A fourth research factor produced an eigenvalue 
greater than 1 . 0 and accounted for 6.1% of variance • 
This factor however loaded saliently (.71) on only one 
pure (correlated with only one factor) rating variable, 
item 14 (instructor explains grading system) . Moderate 
loadings (above .40) also occurred on three complex 
(correlated with more than one factor) rating items: 2 
(instructor well prepared), 1 (clear understanding of 
topics) , and 3 (presents subject matter clearly) . 
Factor IV, a unique construct associated with ratings 
of the research methods course was named 
"thoroughness. " 
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Discussion 

After evaluating results of the factor analyses, 
the authors conclude that somewhat different criteria 
are used by students rating instructional effectiveness 
in the educational research methods course than are 
used by student raters in other graduate education 
courses. The presence of a fourth factor, 

"thoroughness", and the realignment of several items 
loading on Factors I and II provide the basis for this 
conclusion. Factor I accounts for considerably more 
variation among "other course" ratings than among 
research ratings (59,1% vs, 49.5%) while Factor II is 
stronger among the research ratings as indicated by the 
greater explained variance (12,1% vs 7.8%), This 
resulted from the previously mentioned realignment of 
loadings and led the authors to interpret the second 
factor very differently for the research ratings. The 
fourth research factor is not clearly defined by items 
included in the student rating instrument used in this 
study. It loads on only one pure rating item (14) and 
is salient ly related to three items (1, 2, anc 3) all 
of which load on other factors. Inspection of these 
four item loadings however provides rationale for a 
rating dimension which is interpretable and explains 
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essentially as much variance as the third factor 
(interpersonal skills) . 

The seemingly related constructs of "thoroughness" 
and "instructional clarity" appear to be standards used 
by students to judge the effectiveness of instruction 
in a master's level educational research methods 
course. Anxiety and discomfort experienced by students 
T'/ho suddenly find themselves in an unfamiliar 
quantitative environment can perhaps best be exorcised 
by an instructor who communicates course requirements 
and grading policies clearly and who is thorough and 
methodical in his or her approach to instruction. 
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